Repository navigation
fix(occurrence-data): cross-replica generation lock via Redis; tolera… - #960
Merged
Merged
Conversation
…te missing staging files HA sync: the API runs with 2 replicas sharing FULL_OCCURRENCE_DATA_FOLDER on a PVC, and the in-process inflight lock did not stop a second replica from spawning its own OccurrenceGeoJob. kickOff now acquires a Redis lock (SET NX PX, token-checked release via Lua) before spawning the CLI child; background refreshes skip if another replica holds the lock, cold starts wait for it and then re-check staleness. status() reports refreshing from the lock too. All Redis calls are wrapped in a 3s-timeout guard so a Redis outage can never hang the endpoint. ioredis added as a direct dependency. Verified locally: with the lock key held manually, a stale request served in 87ms with X-Occurrence-Data-Refresh: in-progress and spawned no geojob (status showed refreshing:true); after deleting the key the next request spawned the job, which completed and released the lock. Geojob: atomicReplaceFile now checks the staging file exists and skips (with a warning) instead of failing hard on ENOENT, and the staging-cleanup unlink no longer warns on ENOENT (moveFile already removed the file). Reproduced empty/missing target directory runs — both complete successfully. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <[email protected]>
…the Redis lock CI npm ci failed with ERESOLVE: [email protected] declares peerOptional ioredis@^5.0.4 while the lock had ioredis@6. Downgrade to ^5.11.1 (the lock API used — set NX PX / eval / exists / quit — is identical in v5); npm ci now resolves cleanly and nest build passes. Verified at runtime: with the generation lock key held, status reports refreshing:true in ~13ms and a stale request serves with X-Occurrence-Data-Refresh: in-progress while logging 'generation already running on another replica, skipping'; after releasing the key the next request spawns the job and completes cleanly. docs/occurrence-data-pipeline.md updated with the cross-replica Redis lock semantics (background skip vs cold-start wait, lock TTL, 3s Redis guard). Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HA sync: the API runs with 2 replicas sharing FULL_OCCURRENCE_DATA_FOLDER on a PVC, and the in-process inflight lock did not stop a second replica from spawning its own OccurrenceGeoJob. kickOff now acquires a Redis lock (SET NX PX, token-checked release via Lua) before spawning the CLI child; background refreshes skip if another replica holds the lock, cold starts wait for it and then re-check staleness. status() reports refreshing from the lock too. All Redis calls are wrapped in a 3s-timeout guard so a Redis outage can never hang the endpoint. ioredis added as a direct dependency.
Verified locally: with the lock key held manually, a stale request served in 87ms with X-Occurrence-Data-Refresh: in-progress and spawned no geojob (status showed refreshing:true); after deleting the key the next request spawned the job, which completed and released the lock.
Geojob: atomicReplaceFile now checks the staging file exists and skips (with a warning) instead of failing hard on ENOENT, and the staging-cleanup unlink no longer warns on ENOENT (moveFile already removed the file). Reproduced empty/missing target directory runs — both complete successfully.